Avoid double-counting OpenRouter callModel metrics#2005
Avoid double-counting OpenRouter callModel metrics#2005Ronald Koh (ronaldkohhh) wants to merge 2 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6c954e3026
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| const metrics = | ||
| tracedTurnCount === 0 | ||
| ? aggregateOpenRouterCallModelMetrics(rounds, finalResponse) | ||
| : undefined; |
There was a problem hiding this comment.
When callModel traces child turns, this branch now omits metrics from the parent span, but the checked-in OpenRouter e2e contract snapshots still expect metric_keys on openrouter.callModel (for example e2e/scenarios/openrouter-instrumentation/__snapshots__/openrouter-v0911.span-events.json lists prompt_tokens, tokens, and cost for that span). The openrouter-call-model-operation scenario exercises the tool-turn path, so pnpm run test:e2e will fail snapshot comparison until those snapshots/assertions are updated with this intentional trace contract change.
Useful? React with 👍 / 👎.
2 participants
Summary
callModelparent spans when child turn spans were emitted.turn_count, while leaving per-turn usage onopenrouter.beta.responses.sendspans.Why
OpenRouter
callModeltraces currently include usage metrics on both the aggregate parent span and the nested per-turn LLM spans. Span-level token and cost monitors then sum both levels and can report roughly double the actual usage for single-turn calls.When no child turn spans are emitted, the parent still records aggregate metrics so usage is not dropped for simpler
getResponse()flows.Validation
pnpm exec prettier --write js/src/instrumentation/plugins/openrouter-agent-plugin.ts js/src/instrumentation/plugins/openrouter-plugin.ts js/src/instrumentation/plugins/openrouter-agent-plugin.test.ts js/src/instrumentation/plugins/openrouter-plugin.test.tspnpm exec vitest run src/instrumentation/plugins/openrouter-agent-plugin.test.ts src/instrumentation/plugins/openrouter-plugin.test.tspnpm run check:typings